Human-automation collaborative tracker of fused object

ABSTRACT

A system includes a control station that enables efficient human collaboration with automated object tracking. The control station is communicatively coupled to an aerial vehicle to receive full motion video of a ground scene taken by an airborne sensor of the aerial vehicle. The control station spatially registers features of a movable object present in the ground scene and determines motion of the movable object relative to the ground scene. The control station predicts a trajectory of the movable objective relative to the ground scene. The control station tracks the movable object based on data fusion of: (i) the spatially registered features; (ii) the determined motion; and (iii) the predicted trajectory of the movable object. The control station presents a tracking annotation and a determined confidence indicator for the tracking annotation on a user interface device to facilitate human collaboration with object tracking.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. § 119(e)to: (i) U.S. Provisional Application Ser. No. 62/658,121 entitled “AFused Object Tracking Algorithm,” filed 16 Apr. 2018; and (ii) U.S.Provisional Application Ser. No. 62/820,465 entitled “Human-AutomationCollaborative Tracker of Fused Object”, filed 19 Mar. 2019, the contentsof both of which are incorporated herein by reference in their entirety.

ORIGIN OF THE INVENTION

The invention described herein was made by employees of the UnitedStates Government and may be manufactured and used by or for theGovernment of the United States of America for governmental purposeswithout the payment of any royalties thereon or therefore.

BACKGROUND 1. Technical Field

The present disclosure generally relates to testing apparatus andmethods of presenting and tracking moving objects within visual orinfrared imagery.

2. Description of the Related Art

A common Remotely Piloted Aircraft (RPA) Intelligence, Surveillance, andReconnaissance (ISR) mission is to collect Full Motion Video (FMV) ofareas and persons of interest to develop patterns of life whereterrorist and insurgent groups operate. These missions can alsotransition to time sensitive strikes of ground targets. One of the mostfrequent, important, and difficult components of pattern of life andstrike missions is tracking the location and activities of a vehicle(e.g., often a passenger car, pickup, motorcycle). Vehicles move quicklyand can be confused with other vehicles nearby. They may also disappearbehind a building or tree line and be difficult to reacquire once theyreemerge in the FMV. The limited field of view of FMV sensors alsocontributes to the challenge of keeping up with a vehicle. Losing avehicle of high importance is such great concern that in one RPAOperations Center a prominent sign was seen on the wall which displayedthe number of days since the last time a vehicle was lost during amission.

BRIEF SUMMARY

In one aspect, the present disclosure provides a method of enablingefficient human collaboration with automated object tracking. In one ormore embodiments, the method includes receiving full motion video of aground scene taken by an airborne sensor. The method includes spatiallyregistering features of a movable object present in the ground scene.The method includes determining motion of the movable object relative tothe ground scene. The method includes predicting a trajectory of themovable objective relative to the ground scene. The method includestracking the movable object based on data fusion of: (i) the spatiallyregistered features; (ii) the determined motion; and (iii) the predictedtrajectory of the movable object. The method includes presenting atracking annotation on a user interface device. The method includesdetermining a confidence value of the tracking of the movable object.The method includes presenting a confidence indicator on the userinterface device to facilitate human collaboration with object tracking.

In another aspect, the present disclosure provides a system that enablesefficient human collaboration with automated object tracking. In one ormore embodiments, the system includes an aerial vehicle having anairborne sensor. The system includes a control station communicativelycoupled to the aerial vehicle to: (a) receive full motion video of aground scene taken by the airborne sensor; (b) spatially registerfeatures of a movable object present in the ground scene; (c) determinemotion of the movable object relative to the ground scene; (d) predict atrajectory of the movable objective relative to the ground scene; (e)track the movable object based on data fusion of: (i) the spatiallyregistered features; (ii) the determined motion; and (iii) the predictedtrajectory of the movable object; (f) present a tracking annotation on auser interface device; (g) determine a confidence value of the trackingof the movable object; and (h) present a confidence indicator on theuser interface device to facilitate human collaboration with objecttracking.

In an additional aspect, the present disclosure provides a controlstation that enables efficient human collaboration with automated objecttracking. In one or more embodiments, the control station includes anetwork interface communicatively coupled to an aerial vehicle having anairborne sensor. The control station includes a controllercommunicatively coupled to the network interface. The control stationenables the control station to: (a) receive full motion video of aground scene taken by the airborne sensor; (b) spatially registerfeatures of a movable object present in the ground scene; (c) determinemotion of the movable object relative to the ground scene; (d) predict atrajectory of the movable objective relative to the ground scene; (e)track the movable object based on data fusion of: (i) the spatiallyregistered features; (ii) the determined motion; and (iii) the predictedtrajectory of the movable object; (f) present a tracking annotation on auser interface device; (g) determine a confidence value of the trackingof the movable object; and (h) present a confidence indicator on theuser interface device to facilitate human collaboration with objecttracking.

The above summary contains simplifications, generalizations andomissions of detail and is not intended as a comprehensive descriptionof the claimed subject matter but, rather, is intended to provide abrief overview of some of the functionality associated therewith. Othersystems, methods, functionality, features and advantages of the claimedsubject matter will be or will become apparent to one with skill in theart upon examination of the following figures and detailed writtendescription.

BRIEF DESCRIPTION OF THE DRAWINGS

The description of the illustrative embodiments can be read inconjunction with the accompanying figures. It will be appreciated thatfor simplicity and clarity of illustration, elements illustrated in thefigures have not necessarily been drawn to scale. For example, thedimensions of some of the elements are exaggerated relative to otherelements. Embodiments incorporating teachings of the present disclosureare shown and described with respect to the figures presented herein, inwhich:

FIG. 1 is a diagram of an intelligence, surveillance, and reconnaissance(ISR) and targeting system, according to one or more embodiments;

FIG. 2 is a diagrammatic illustration of an exemplary hardware andsoftware environment of a UAV control station, according to one or moreembodiments;

FIG. 3 is a flow diagram of an example framework of an object tracker,according to one or more embodiments;

FIG. 4 is an example video image of a normal tracking situation,according to one or more embodiments;

FIG. 5 is an example of an example video image having a viewpoint changeof the object of FIG. 4, according to one or more embodiments;

FIG. 6 is an example video image having an illumination change of theobject of FIG. 5, according to one or more embodiments;

FIG. 7 is an example video image of a full occlusion of the object bytrees, according to one or more embodiments;

FIG. 8 is an example video image of a failed image registration event,according to one or more embodiments;

FIG. 9 is a depiction of a user interface having an example video imagewith object tracker confidence information indicating high confidence,according to one or more embodiments;

FIG. 10 is a depiction of a user interface having an example video imagewith object tracker confidence information indicating medium confidence,according to one or more embodiments;

FIG. 11 is a depiction of a user interface having an example video imagewith object tracker confidence information indicating low confidence,according to one or more embodiments;

FIG. 12 is a state diagram that illustrates example user interface (UI)presentations respectively for “not tracking”, “manual”, and “auto”modes of object tracker of FIG. 3, according to one or more embodiments;

FIG. 13 is a state diagram that illustrates UI presentations having anindicator box which states “Tracker Unavailable” or “Tracker Available”based on whether the image registration process was successful,according to one or more embodiments;

FIG. 14 is a state diagram illustrating UI presentations that show thedifferent tracking states when Preview Mode is on, according to one ormore embodiments; and

FIG. 15 is a flow diagram illustrating method of enabling efficienthuman collaboration with automated object tracking, according to one ormore embodiments.

DETAILED DESCRIPTION

According to aspects of the present innovation, a fused object trackingmethod, system and computer program product facilitates human-automationcollaborative tracking by communicating confidence and tracking faultsthrough a user interface.

FIG. 1 is a diagram of an intelligence, surveillance, and reconnaissance(ISR) and targeting system 100. The system 100 includes one or moreunmanned aerial vehicles (UAVs) 102 that provide full motion video (FMV)104 of a movable ground target 106 to a collaborative human-automationtracking system 108 that includes a UAV command station 110. Anautomated video object tracker (“tracker”) 112 enhances and improves theUAV command station 110 by providing humanly-perceptible indicators 114and control affordances 116 on one or more user interface devices 118 toa human operator 120. The indicators 116 enhance situational awarenessof the human operator 120 by indicating historical path 122 of themovable ground target 106 and a confidence level indication 124 in theautomated tracking. An operator can visually extrapolate the historicalpath 122 to determine an expected path. In one or more embodiments, whenin automated control, the tracker 112 communicates a sensor steeringcommand 126 directly to an autonomous controller 128 of a UAV 102 thatprovides the FMV 104 or a remotely piloted aircraft (RPA) controlstation 130. In one or more embodiments, when in automated control, thetracker 112 communicates the sensor steering command 126 indirectly toan autonomous controller 128 of a UAV 102 that provides the FMV 104 or aremotely piloted aircraft (RPA) control station 130.

Optical object tracking algorithms have been deeply researched for fullmotion video (FMV) and wide area motion imagery (WAMI). Optical objecttracking algorithms are designed to follow pedestrians and vehicles ofinterest as viewed by these sensors. Object trackers are selectivelyused by the US government to perform intelligence, surveillance, andreconnaissance (ISR) and targeting within the U.S. Department of Defense(DoD) and other Federal agencies. Likewise, foreign government agenciesutilize them for similar purposes. Optical object tracking algorithmsare also a component of commercial imagery analysis software.

In principle, object trackers can relieve human observers ofcontinuously attending to the location of the object within the sensorimagery. However, fully automated tracking is still unrealized, asobject trackers can lose or confuse tracks over the myriad of lightingand viewing conditions where the automated tracking may be deployed. Forthe foreseeable future, human observers will continue to play animportant role in the tracking function. Instead of perfecting fullyautomated tracking, the present disclosure aims to improve collaborationbetween human observers and automated object trackers.

Due to object tracker limitations, delegating tracking functions betweenhumans and automation is a dynamic process which depends on the qualityof the imagery and viewing conditions. Judgements about delegation to anobject tracker require assessment of object tracker performance.Existing object trackers do not communicate dynamic self-assessments ofperformance or tracking confidence to a user. This makes it difficultfor a user to calibrate trust in the object tracker and make appropriatetracking delegation decisions from moment to moment during an objecttracking event.

In addition, when tracking faults occur, existing object trackers do notalert the user they have ceased tracking. When an object tracker stopsfollowing an object, a box drawn around the designated object simplydisappears. In addition, there is no information provided about why thetracking failure occurred.

According to aspects of the present disclosure, object trackingalgorithms include important capabilities lacking in existing algorithmsto enable dynamic collaboration with a user: (i) Tracker Fusion &Confidence Reporting; (ii) Scene Characterization & Fault Diagnosis; and(iii) Graphical User Interfaces with Alerting.

Tracker Fusion & Confidence Reporting: The proposed algorithms fusethree types of tracking modalities (features, motion, and motionestimation) and provide confidence values for feature tracking, motiontracking, and also image registration. These confidence values arenormalized along a common scale and are used to coordinate weightingsassigned to each tracker input. The confidence values are also used tocommunicate to the user the relative ability of the object trackingalgorithm to continue following an object without user assistance. Lowconfidence situations trigger alerts to the user that object tracking isfaltering. Tracker fusion of feature, motion, motion estimation, andassociated confidence values, are implemented within a softwareprototype demonstrated with full motion video. At present, the imageregistration confidence measures are not implemented within a softwareprototype.

Scene Characterization & Fault Diagnosis: The proposed algorithms alsomonitor the object and scene for changes in viewpoint, illumination,partial or full occlusions, a lost track, and image registrationfailures. These changes and faults are communicated to the user incombination with tracking confidence information, and low confidencealerts often correlate with scenery changes. Thus the user is informednot only of approaching or actual tracking faults, but also the possiblecauses. Detections of lost tracks and image processing failures arerepresented within the software prototype.

Graphical User Interfaces with Alerting: The object tracker confidencemeasures and fault diagnosis are communicated to the user through anintuitive graphical user interface. The current confidence level iscommunicated in several ways along with notifications of tracking faultswhen they occur. The user is alerted to low confidence and fault statesthrough a combination of visual and auditory alerts. The user is alsoable to preview the object tracker performance before delegating thetracking function to the automation. At present the user interfaceprovides limited diagnosis of tracking faults.

In a context of the U.S. Air Force (USAF), the proposed object trackinginnovations assist ISR collection and analysis activities. At least twopossible users are envisioned, sensor operators and geospatialintelligence analysts. Using the proposed object tracker, sensoroperators are relieved at times of continually steering a sensor tomaintain line of sight with a moving object. Instead, the sensor starepoint would be yoked to the object location as specified by the objecttracker. This would free attention resources of sensor operators forother important activities such as anticipating object movements andconfiguring sensors to provide the desired picture. Likewise,intelligence analysts whose function is to watch and report on relevantscene activity could be relieved of visually tracking a single targetobject and devote more resources to interpreting the behavior of themoving object as well as others in the broader scene context. Theproposed object tracking algorithms could be combined with mappingapproaches to further automate the generation of vehicle stop reportswhich are a standard product of intelligence analysts in a DistributedGround System (DGS).

A common Remotely Piloted Aircraft (RPA) Intelligence, Surveillance, andReconnaissance (ISR) mission is to collect Full Motion Video (FMV) ofareas and persons of interest to develop patterns of life whereterrorist and insurgent groups operate. These missions can alsotransition to time-sensitive strikes of ground targets. One of the mostfrequent, important, and difficult components of pattern of life andstrike missions is tracking the location and activities of a vehicle(e.g., often a passenger car, pickup, motorcycle). [1]; [2]; and [3]

Human Teamwork with Vehicle Tracking: A distributed team of USAFpersonnel is responsible for the common task of tracking objects ofinterest. In the RPA Ground Control Station (GCS) are the RPA Pilot andthe Sensor Operator (SO) who work together to ensure the vehicle or areaof interest remains in view in the FMV. The SO is tasked withconfiguring and steering the sensor to keep the vehicle of interest inview. The SO also ensures the desired image is achieved (magnification,selection of electro-optical (EO) or infrared (IR) mode, look angle,etc.) for the consumers of the FMV who are the Intel Analysts in a DGS.The DGS FMV Analysts are officially charged with interpreting anddocumenting ground activity. See TABLE 1 below. Nonetheless, the SOinformally assists the FMV Analysts by clarifying the identity andactivity of actors.

There are several DGS FMV Analysts assigned to each FMV feed and theyrotate the role called “Eyes On” who continually observes the sensorview and verbalizes relevant events on the ground (termed “call outs”).Other DGS team members capture images of observed activity and produceintelligence products for the customer who requested the FMV mission.[1]

Distributed Ground Station (DGS) FMV Remotely Piloted Aircraft (RPA)Analyst Look for potential targets Observes target(s) Steer sensor totrack target, coordinating Calls-out target events with RPA pilot, DGS,supported unit Clarify identity of targets with DGS Manages significantevent time log Achieve desired EO/IR image with DGS Captures images forforensic analysis Continuously observes FMV Continuously observes FMVConducts long duration missions Conducts long duration missions

The SO and FMV Analyst share a common human performance challenge: Theyboth continually observe the FMV feed for missions which can last hours.Continually tracking vehicles in complex terrain or in the presence ofsimilar objects over long periods is fatiguing. The SO and FMV Analystmay miss relevant activity or lose visual lock due to a vigilancedecrement. [4] The SO is a particularly critical tracking team memberbecause the FMV feed arrives from the RPA several seconds sooner and isof higher quality than the FMV received by the DGS. Thus the SO is thebest equipped team member to visually track the vehicle.

To address these human performance challenges with following vehicles,the USAF employs several tactics. In the DGS the Eyes On position isrotated with another FMV analyst after a prescribed period of 30 or 60minutes. [1]; and [5]. The DGS is also able to rewind the recent FMVrecording to perform forensic analysis of recent activity, such as whenan object of interest is lost. During strike missions or if the vehicleis a high value target, multiple USAF personnel at the unmanned aerialvehicle (UAV) operations center and DGS site may watch the same sensorfeed. This redundancy of observers protects against the possibility ofany one or two people losing sight of the vehicle or other high valuetarget. Another strategy employed by these observers is to physicallytrace the object on the FMV feed with a finger or pencil.

Automated Tracking Technology: There is automation technology to assistwith tracking objects seen in the FMV. Optical object trackers are acomputer vision application designed to detect and visually followobjects to increase visual saliency. Some generally-known objecttrackers are triggered by motion cues and will automatically draw asemi-transparent, boxlike symbol called a “bounding box” around anymoving object. (For a more in-depth description of object trackingapproaches, see [6]. This type of tracker does not usually maintainidentity information about the object, so that if the object disappearsand reappears in the view, the object tracker does not recognize it asthe same object seen before. Other object trackers analyze theappearance of the object in order to establish a unique signature, suchthat the object may be distinguished from other objects and bereacquired if the object temporarily disappears from view. When usingthis type of object tracker, the SO specifies which object to watch byexpanding a bounding box symbol around the object.

Object trackers can sometimes compensate for limitations with humanvisual attention. Once locked onto an object, the automated tracker canserve as another observer of the object proving an additional line ofdefense against a lost target. The bounding box provides a persistentvisual indicator of the object location within the FMV feed that is seenby both the SO and the DGS crew. Although more commonly used by SensorOperators, FMV Analysts in DGS could also employ object trackers to helpbuild vehicle routing diagrams. The object trackers can also beconfigured to automatically capture images when the object performscertain behaviors such as stop, go, and turn.

In practice, however, object tracker technology is not nearly as robustas human vision in maintaining visual lock on objects such as cars orpeople in FMV. Generally-known object trackers can lose visual lock dueto complexities in the environment such as object density, occlusions,clouds, smoke and haze. Object trackers are very sensitive to the imagequality and FMV from RPAs can provide highly variable image quality.Intermittent image rates, blurring, smearing and other imagery anomaliescan cause the object tracker to lose visual lock and preventreacquisition. Finally, generally-known object trackers are alsochallenged by the ways the SO uses the sensor. Switches between daytime(color) imagery and infrared (IR) imagery, changes in magnification ofthe view, and dramatic slewing of the sensor can all disrupt the abilityof the object tracker to visually follow a designated object.

Currently, object trackers are also poorly designed as a team member.The user must continually monitor the automation performance because itdoes not alert when it is struggling with processing imagery or hasfailed (i.e., no longer tracking the designated object). The onlyfeedback given to the user is when the bounding box either moves off ofthe target or simply vanishes. Furthermore, current object trackers donot report diagnostic information during or after a tracking failurethat might assist the user with understanding the competency boundariesof the automation. For example, object trackers could provide the userwith diagnostic information that a “partial occlusion” is occurring asthe designated target moves behind a row of trees along the road.

The inadequate amount of object tracker performance feedback leaves theuser ill-equipped to appropriately calibrate trust and utilize theautomation, often resulting in over- and under-reliance. While the usermay intuit the competency boundaries of the automated tracker overextended situations of use, the momentary and immediate futureperformance of the tracker is opaque to the user. As a result, the useris challenged in making real time decisions about when the tracker canbe relied upon to perform the tracking task. Without extensive trainingor experience with the object tracker, the user will be unaware of thereliability of the tracker under the current visual conditions. Thisreduces the potential usefulness of the object tracker as a relief aidwhen the human observer is fatigued.

Automation confidence is often included in frameworks of automationtransparency, where transparency is a method for establishing sharedawareness and shared intent between a human and a machine. [8]; [9]; and[10]. The Chen et al. model is unique in including temporal component ofconfidence information. Automation should not only communicate thepresent level of confidence in performance, but also the historical andprojected future confidence in performance.

The present disclosure includes enhancements to object trackingtechnology so that more effective human-automation teaming is achieved.A primary advancement is object tracker machine confidence measures. Theobject tracker generates estimates of its ability to continue trackingthe designated object of interest. Reporting self-assessment orautomation confidence information may help a sensor operator betterunderstand the automation performance and assess when the object trackercan be trusted and relied upon. Another advancement is automationdetection and reporting by the object tracker of changes includingobject viewpoint, illumination, occlusions, image registration failures,and lost tracks. A further innovation is an intuitive user interface tocommunicate tracking confidence and tracking failures through salientvisual and auditory displays. Each of these innovations is described indetail in the next section.

Tracker Fusion & Confidence Reporting: An object tracking framework wasdeveloped to create more robust automated tracking performance. Theframework was developed to parallel the functions of the human visualsystem during object following events.

The human perceptual system influences the detection and tracking ofvehicles. Humans recognize specific objects based on their visualfeatures (contours, edges, colors, etc.), providing the ability todetect and track those features. This type of tracking works well whenthe features are unchanged during tracking, but in many scenarios, thefeatures begin to change as the object moves through an environment andis viewed from different perspectives and different light angles andlevels. The object also undergoes external changes as well, like partialocclusions. When the features begin to degrade, we must utilize adifferent tracking modality that does not depend on the features. Thismodality is motion, which is detected based on differences between thebackground and object foreground. Objects in our periphery are usuallydetected based on the motion cues of the object rather than thefeatures. Once the alerted human focuses on the object, then featuresare obtained for detection and tracking. Motion and feature informationprovide complimentary cues for object detection and tracking. Motiondetection does degrade during occlusion scenarios where some or all ofthe motion cues are absent, so another type of modality must beimplemented. This modality is motion estimation, which utilizes theprevious information about the track of the object to determine thefuture location of the object. When an object goes under an occlusion,the human perceptual system assumes the speed and direction of theobject may continue without much change to these parameters. Byestimating the speed and direction parameters we can estimate the objectlocation moment to moment when it is not visible, then verify the objectidentity when the object motion or features are visible again at thepredicted location.

FIG. 2 is a diagrammatic illustration of an exemplary hardware andsoftware environment of a UAV control station 200, such as UAV controlstation 110 (FIG. 1), customized to implement a collaborativehuman-automation tracking and targeting system (CHATTS) controller 202consistent with embodiments of the innovation. UAV control station 200is in part a customized information handling system (IHS) 100 thatperforms at least a part of the methodologies and features as describedherein. UAV control station 200 can include processing resources forexecuting machine-executable code, such as a central processing unit(CPU), a programmable logic array (PLA), an embedded device such as aSystem-on-a-Chip (SoC), or other control logic hardware. UAV controlstation 200 can also include one or more computer-readable medium forstoring machine-executable code, such as software or data. Additionalcomponents of UAV control station 200 can include one or more storagedevices that can store machine-executable code, one or morecommunications ports for communicating with external devices, andvarious input and output (I/O) devices, such as a keyboard, a mouse, anda video display. UAV control station 200 can also include one or moreinterconnects or buses operable to transmit information between thevarious hardware components.

UAV control station 200 includes processors 204 and 206, chipset 208,memory 210, graphics interface 212, a basic input and outputsystem/extensible firmware interface (BIOS/EFI) module 214, diskcontroller 216, hard disk drive (HDD) 218, optical disk drive (ODD) 220,disk emulator 222 connected to an external solid state drive (SSD) 224,input/output (I/O) interface (I/F) 226, one or more add-on resources228, a trusted platform module (TPM) 230, network interface 232,management block 234, and power supply 236. Processors 204 and 206,chipset 208, memory 210, graphics interface 212, BIOS/EFI module 214,disk controller 216, HDD 218, ODD 220, disk emulator 222, SSD 224, I/Ointerface 226, add-on resources 228, TPM 230, and network interface 232operate together to provide a host environment of UAV control station200 that operates to provide the data processing functionality of theinformation handling system. The host environment operates to executemachine-executable code, including platform BIOS/EFI code, devicefirmware, operating system code, applications, programs, and the like,to perform the data processing tasks associated with UAV control station200.

In a host environment, processor 204 is connected to chipset 208 viaprocessor interface 238, and processor 206 is connected to the chipset208 via processor interface 240. Memory 210 is connected to chipset 208via a memory bus 242. Graphics interface 212 is connected to chipset 208via a graphics bus 244, and provides a video display output 246 tographical display(s) 248. In a particular embodiment, UAV controlstation 200 includes separate memories that are dedicated to each ofprocessors 204 and 206 via separate memory interfaces. An example ofmemory 210 includes random access memory (RAM) such as static RAM(SRAM), dynamic RAM (DRAM), non-volatile RAM (NV-RAM), or the like, readonly memory (ROM), another type of memory, or a combination thereof.

BIOS/EFI module 214, disk controller 216, and I/O interface 226 areconnected to chipset 208 via an I/O channel 250. An example of I/Ochannel 250 includes a Peripheral Component Interconnect (PCI)interface, a PCI-Extended (PCI-X) interface, a high speed PCI-Express(PCIe) interface, another industry standard or proprietary communicationinterface, or a combination thereof. Chipset 208 can also include one ormore other I/O interfaces, including an Industry Standard Architecture(ISA) interface, a Small Computer Serial Interface (SCSI) interface, anInter-Integrated Circuit (I2C) interface, a System Packet Interface(SPI), a Universal Serial Bus (USB), another interface, or a combinationthereof. BIOS/EFI module 214 includes BIOS/EFI code operable to detectresources within UAV control station 200, to provide drivers for theresources, initialize the resources, and access the resources. BIOS/EFImodule 214 includes code that operates to detect resources within UAVcontrol station 200, to provide drivers for the resources, to initializethe resources, and to access the resources.

Disk controller 216 includes a disk interface 252 that connects the diskcontroller to HDD 218, to ODD 220, and to disk emulator 222. An exampleof disk interface 252 includes an Integrated Drive Electronics (IDE)interface, an Advanced Technology Attachment (ATA) such as a parallelATA (PATA) interface or a serial ATA (SATA) interface, a SCSI interface,a USB interface, a proprietary interface, or a combination thereof. Diskemulator 222 permits SSD 224 to be connected to UAV control station 200via an external interface 254. An example of external interface 254includes a USB interface, an IEEE 1394 (Firewire) interface, aproprietary interface, or a combination thereof. Alternatively,solid-state drive 164 can be disposed within UAV control station 200.

I/O interface 226 includes a peripheral interface 256 that connects theI/O interface to add-on resource 228, to TPM 230, and to networkinterface 232. Peripheral interface 256 can be the same type ofinterface as I/O channel 242, or can be a different type of interface.As such, I/O interface 226 extends the capacity of I/O channel 242 whenperipheral interface 256 and the I/O channel are of the same type, andthe I/O interface translates information from a format suitable to theI/O channel to a format suitable to the peripheral channel 256 when theyare of a different type. Add-on resource 228 can include a data storagesystem, an additional graphics interface, a network interface card(NIC), a sound/video processing card, another add-on resource, or acombination thereof. Add-on resource 228 can be on a main circuit board,on separate circuit board or add-in card disposed within UAV controlstation 200, a device that is external to the information handlingsystem, or a combination thereof.

Network interface 232 represents a NIC disposed within UAV controlstation 200, on a main circuit board of the information handling system,integrated onto another component such as chipset 208, in anothersuitable location, or a combination thereof. Network interface 232includes network channels 258 and 260 that provide interfaces to devicesthat are external to UAV control station 200. In a particularembodiment, network channels 258 and 260 are of a different type thanperipheral channel 256 and network interface 232 translates informationfrom a format suitable to the peripheral channel to a format suitable toexternal devices. An example of network channels 258 and 260 includesInfiniB and channels, Fibre Channel channels, Gigabit Ethernet channels,proprietary channel architectures, or a combination thereof. Networkchannels 258 and 260 can be connected to external network resources (notillustrated). The network resource can include another informationhandling system, a data storage system, another network, a gridmanagement system, another suitable resource, or a combination thereof.In an exemplary embodiment, network channel 258 is communicativelycoupled to UAV 262 to directly adjust slaving of a sensor 264 or toperform other targeting. Network channel 260 is communicatively coupledto RPA control station 266 to indirectly adjust slaving of the sensor264 or to perform other targeting.

Management block 234 represents one or more processing devices, such asa dedicated baseboard management controller (BMC) System-on-a-Chip (SoC)device, one or more associated memory devices, one or more networkinterface devices, a complex programmable logic device (CPLD), and thelike, that operate together to provide the management environment forUAV control station 200. In particular, management block 234 isconnected to various components of the host environment via variousinternal communication interfaces, such as a Low Pin Count (LPC)interface, an Inter-Integrated-Circuit (I2C) interface, a PCIeinterface, or the like, to provide an out-of-band (OOB) mechanism toretrieve information related to the operation of the host environment,to provide BIOS/UEFI or system firmware updates, to managenon-processing components of UAV control station 200, such as systemcooling fans and power supplies. Management block 234 can include anetwork connection to an external management system, and the managementblock can communicate with the management system to report statusinformation for UAV control station 200, to receive BIOS/UEFI or systemfirmware updates, or to perform other task for managing and controllingthe operation of UAV control station 200. Management block 234 canoperate off of a separate power plane from the components of the hostenvironment so that the management block receives power to manage UAVcontrol station 200 when the information handling system is otherwiseshut down. An example of management block 234 may include a commerciallyavailable BMC product that operates in accordance with an IntelligentPlatform Management Initiative (IPMI) specification. Management block234 may further include associated memory devices, logic devices,security devices, or the like, as needed or desired.

Within memory 210, HDD 218, ODD 220, or SSD 224, one or more softwareand/or firmware modules and one or more sets of data can be stored thatcan be utilized during operations of UAV control station 200. These oneor more software and/or firmware modules can be loaded into memory 210during operation of UAV control station 200. Specifically, in oneembodiment, memory 210 can include therein a plurality of such modules,including an object tracker application 268, one or more otherapplications 270, operating system (OS) 272, and data 274. Thesesoftware and/or firmware modules have varying functionality as disclosedherein when their corresponding program code is executed by processors204, 206.

FIG. 3 is a flow diagram of an example framework of object tracker 300.Several processes within the human visual processing system areparalleled in a novel object tracking framework. The framework 300contains five (5) different algorithms that provide robust objecttracking: (i) image registration 302; (ii) Kalman motion predictiontracking 304; (iii) moving object detection (MOD) 306; (iv) directionalringlet intensity feature transform (DRIFT) 308; and (v) tracker fusion310.

The first algorithm that is used for object tracking is imageregistration 302, which stabilizes video for the rest of the algorithms.[11]; [12]; [13]; and [14]. The last algorithm is tracker fusion 310,which combines Kalman tracking, MOD, and DRIFT to find the best track ofthe vehicle.

Image Registration: Image registration 302 is vital in the detection andtracking of the objects of interest. Observing an object from a mobileplatform causes changes in the image viewpoint, and the location of theobject in one image versus another is different even if the object hasnot moved. In one or more embodiments, image registration 302 includesone or more of: (i) accelerated segment test (FAST) corners 312; (ii)fast retina keypoint (FREAK) features 314; (iii) feature matching 316;(iv) transform estimation 318; and (v) transform imagery 320. Toaccomplish the registration of the imagery and have the imagerystabilized through the imagery, Features from FAST corners 312 aredetected in the image pair, which provide locations of distinct edges ineach image. We then obtain FREAK features 314 based on a difference ofGaussians formed in a retinal orientation. Feature matching 316 matchesthese features in each pair of images using the match distance betweenfeatures. Transform estimation 318 selects the optimal transform.Transform imagery 320 uses the optimal transform to transform the firstimage and all associated points to the image space of the second image.

Image Registration Confidence: For object trackers that utilizehistorical and geospatial information for detection and tracking, imageregistration is a necessary implementation to map the image space tolink the information. The object tracker that has been developedutilizes image registration for historical information and motioninformation, which allows for deeper analysis of the object. With theimage registration working well, many parts of the object trackingsystem work to supply the necessary analysis information. But, there arecases where the image registration fails, which disrupts the historicalinformation linkage and the motion models used. A confidence score wasdeveloped for the image registration algorithm, which will notify theuser of image registration failures. For different types of imageregistration algorithms, a confidence computation will need to bedeveloped for each or the image registration used in our development, ituses a feature point matching scheme to find several matching points inthe image to morph the previous image to the current image space.Therefore, there are three places of information that we can utilizefrom the image registration algorithm to create the confidence score:(i) number of feature points for each image; (ii) number of featurepoint matches; and (iii) in-frame matching.

Number of Feature Points: The image registration algorithm utilized inour object tracker detects various feature points within the image. Tomatch feature points within the image, the algorithm must have a certainnumber of feature points to consider. In many cases, having more featurepoints in the image allows for more potential matches, so we havedeveloped a threshold to determine the number of feature matches neededfor the video. The thresholds can be set for the different sensor typesused for the object detection and tracking scenarios.

${confidence} = {( {1 - \frac{{testFeatures} - {refFeatures}}{testFeatures}} ) \times 100}$

Number of Feature Point Matches: The next stage is to determine if thefound features from the two images are matching. The current algorithmcreates feature vectors for comparison and removes all non-matchingcomponents, leaving only points used for determining the imagetransform. Just like the previous threshold for the number of featurepoints, we have developed a threshold for the feature point matches.

${confidence} = {( {1 - \frac{{testFeaturesMatches} - {refFeaturesMatches}}{testFeaturesMatches}} ) \times 100}$

In-Frame Matching: This phenomena is found when there are matchingfeatures within one single frame, which are usually repeating patternsin video like road lines. The feature matching algorithm may be confusedas to which features to match, so it usually finds the best match of thetwo to match the image. By determining any matching features within theimage, we can determine if the image may get confused in registration,especially in poor registration conditions. The confidence valuecomputation is shown below.

${confidence} = {( \frac{{inFrameFeatures} - ( {{matchFeatures}/2} )}{inFrameFeatures} ) \times 100}$

Methods for combining these confidence scores are still underdevelopment. Once implemented, the image registration confidencereporting will provide users with increased understanding of theperformance envelope of the object tracker under different viewingconditions.

Kalman Motion Prediction Tracking (Kalman Tracker 304): With thedifferent object tracks, we utilize Kalman tracking to estimate thetrajectory of the objects with predict track component 322, which helpswith narrowing the search region for the object and reduces thecomputation time of the system. The estimation of the position andvelocity (two-dimensions) can be found by using state space equations.The result can predict the next position with a constant velocity oreven predict position and velocity with a constant acceleration,depending on the inputs. Due to the dynamic environment of the imagery,point transformations are done to register the position and velocityvectors on the moving imagery to give the best indication of the objectmovement. Kalman tracker 304 also includes correct track component 324discussed below.

Moving Object Detection (MOD) Tracking (MOD 306): MOD 306 includes oneor more of: (i) background subtraction 326; (ii) Otsu's thresholding328; (iii) morphological operations 330; (iv) connected components 332;(v) region grouping 334; and (vi) MOD bounding box 336. Moving objectdetection utilizes the motion from frame to frame to develop thelocation of a specific object. Many types of moving object detectionutilize a background model to find the foreground regions of the image.Once the image regions are found, processing is done to define the pixelregions and connect the regions together to form specific objects thathave motion. There are three types of moving object tracking types: (i)point tracking; (ii) kernel tracking; and (iii) silhouette tracking.Point tracking utilizes a point determined by the region of interest,which is the center of the object. Kernel tracking uses shape andfeatures to determine the tracking ability. Silhouette tracking utilizesshape matching to find the corresponding object.

When images are moving and features are changing, we can ascertain theobserved object is the one of interest due to the object movement. Whenthere is no movement in the object, the region of interest should notchange. Even in high occlusion areas, we can safely assume that anymoving portion in the vicinity of the previously detected object will bepart of the object. Therefore, if we can merge motion detection andfeature detection together, we can supplement information from bothduring the failure stages of the algorithms.

Depending on the video type, a background model could be much harder toobtain, especially with a mobile platform. Background models must beupdated to encompass the image movement and new regions must beaccounted for to fully utilize the benefits of motion tracking. Thealgorithm necessitates the use of image registration for registering thelocations of the image that are not moving, though the imagery ismoving. Therefore, moving object detection requires the accuracy of theregistration method to be fairly high for the background model to besufficiently updated and the foreground features to be highlighted. Weutilize a simple background subtraction due to the mobile nature of theimagery which causes appearance changes from slight miss-registrations.

The moving object detection 306 first obtains an image subtractionbetween the pairs of registered images. Once the difference is obtained,we threshold the difference areas to get all moving objects in thescene. Due to the noise components found in the image registration, wemust remove them with a series of morphological operations. First, weuse ‘closing’ to remove any small movements due to registration errors.Lastly, we use ‘opening’ to merge moving portions that should be mergetogether. Using connected component analysis, we can determine allconnected regions and provide a centroid for the moving region. Due tosome connected components not being fully connected, we can provide anassumption of the search region and size of the object in order toconnect components that are not connected but should be. This identifiesall of the moving objects in the scene and can be used for objectinitialization, partial occlusion detection, and any other type of datathat needs motion detection.

DRIFT Feature Tracker (DRIFT 308): DRIFT 304 includes one or more of:(i) sub-image scan 338; (ii) Gaussian ringlet mask 340; (iii) histogramfeatures 342; (iv) earth mover's distance 344; and (v) DRIFT boundingbox 346. The DRIFT features are able to detect an object of interestwithout any movement information. This feature-based tracker providesthe ability to re-identify objects and continue tracking an objectduring normal conditions. The feature tracker uses Gaussian ringlets topartition the image region of the object for rotation invariance. [15].The different features that can be used are intensity and gradientinformation. The gradient information can be obtained from kirsch masks,which find the directional gradient information from 8 differentdirections. The intensity and gradient information is then grouped inhistograms and concatenated together to be used for evaluation betweenother feature sets. [16]. The comparison is done using earth mover'sdistance (EMD), which compares distributions from the reference and testhistograms from the reference and test image respectively.

Tracker Fusion Based on Confidence: A confidence and speed component 348of the tracker fusion algorithm 310 in use leverages the three types ofobject trackers described above which operate based on different objectimage parameters (DRIFT 308 for object features or appearance cues,Moving Object Detection 306, Kalman Motion Prediction tracking 304 formotion prediction). The DRIFT 308 and Moving Object Detection trackers306 generate confidence values as an approximation of the performance ofthe tracker type. Confidence as defined here is an estimate of thelikelihood the object being tracked is the one designated as the targetby the user. The object tracking confidence values are dynamic andchanges based on the estimated performance of the object tracker whilefollowing a vehicle. Failure mode component 350 of tracker fusion 310determines what type of failure has occurred based on the reportedconfidence levels.

When dealing with feature trackers and moving object detection, thereare changing conditions that can change the confidence in the algorithm.The following equation shows the confidence based on the DRIFT featuretracker.

${confidence} = {( {1 - \frac{{testDistance} - {refDistance}}{testDistance}} ) \times 100}$where refDist is the difference between two reference images in nearframes and testDist is the difference between the reference detectionand the current frame detection. The basis of this equation is on thedifference between features when a new detection is found. Due to nothaving a ground truth for comparison, the only assumption that can bemade is the newly found track should not have a changing distancegreater than an observed change found in the beginning stages. Eventhough this may fail with very small changes found in the beginning, thereference distance can be updated and even a manifold can be createdbased on the confidence found in the imagery.

The following equation shows the confidence based on the Moving ObjectDetection:

${confidence} = {( {1 - \frac{{testArea} - {refArea}}{testArea}} ) \times 100}$where refArea is the reference moving object detection area and testDistis the current frame's moving object detection area. Given that thereare two different confidences that are calculated, we can derive thetotal confidence based on the maximum confidence between both trackers.

The basis of this equation is the difference between features when a newdetection is found. Due to not having a ground truth for comparison, theonly assumption that can be made is the newly found track should nothave a changing distance greater than an observed change found in thebeginning stages. Even though this may fail with very small changesfound in the beginning, the reference distance can be updated and even amanifold can be created based on the confidence found in the imagery.

The normalized confidence values of the DRIFT and Moving ObjectDetection trackers are used to coordinate with each other and the KalmanMotion Prediction tracker, as well as the human user. A binary weightingis applied to each tracker output based on the confidence level of theDRIFT feature and Moving Object Detection trackers. However, fusionalgorithms incorporating non-binary weighting are under development.

Referring to TABLE 2, if the DRIFT feature tracker has a confidencelevel of 80% or greater, the location estimate provided by the DRIFTfeature tracker is given a weight of 1.0 and the other two trackers aregiven a weight of 0.0. If the DRIFT confidence is 79% or less and themoving object detection tracker confidence is 50% or greater, thelocation estimate of the moving object detection tracker receives aweight of 1.0 and the other two trackers receive a weight of 0.0. If theconfidence value of the moving object detection tracker is between 26%and 49% and the DRIFT tracker confidence is 79% or less, the motionprediction tracker estimate of the object location is given a weight of1.0 and the other two trackers receive a weight of 0.0. When the MovingObject Detection confidence drops below 25%, all tracking operationscease.

TABLE 2 Object Tracking Weighting Approach. Kalman Feature MotionFeature Tracker Motion Tracker Tracker Confidence Confidence WeightWeight Weight  >80% NA 1 0 0 <=80%  50% to 100% 0 1 0 NA 26% to 49% 0 01 NA   0 to 25%* 0 0 0 *Stop all tracking processes until confidenceregained

Scene Characterization and Fault Diagnosis: The proposed algorithms alsoself-diagnose the nature of low confidence situations where the trackerperformance is compromised. In the current software implementation, thetwo basic tracking failures are losing the object, and unsuccessfulprocessing of the visual imagery as a whole. These tracking faults arecommunicated to the user in combination with low confidence alerting.

In addition, a diagnosis of the possible reasons for object loss orimage processing failures can also communicated so the user betterunderstands the circumstances that gave rise to the tracking faults. Thetracking algorithms provide the necessary information to not only reportconfidence information, but also reasons for low confidence. Reportingthe estimated causation for low confidence events helps the user developa deeper understanding over time of the performance envelope of theobject tracker under various viewing situations. We have identifiedseveral tracking situations that can be diagnosed using informationcollected and generated by the tracking algorithms: Normal, ViewpointChanges, Illumination Changes, Partial/Full Occlusions, Lost Track, andFailed Image Registration. The following information is used to diagnosethese tracking situations: (i) Feature Confidence; (ii) MotionConfidence; (iii) Image Registration Confidence; (iv) Trajectory; and(v) Mean Color.

For each tracking situation or mode, we will describe how thisinformation is used in combination to diagnose the situation.

Normal: During ideal operation of the object tracker, the variouscomponents should process normally resulting in relatively unchangedconfidence values. Normal tracking situations are diagnosed when featuredetection and motion detection confidence is high. When an object isbeing tracked, the features do not normally change drastically and areusually moving in the same direction.

During a normal tracking mode, the user can safely assume that thetracker is performing well and the user can trust the object tracker tofollow the designated object. This mode generally operates in clearvisual conditions without any obstruction of the object. We can create athreshold for the confidence values as to when the object track leavesthe normal mode due to changing conditions. This threshold could beadjusted by the user. The characterization of a normal scene is shown inthe TABLE 3 below.

TABLE 3 Information Type Characterization Feature Confidence HighConfidence Motion Confidence High Confidence Image RegistrationConfidence High Confidence Trajectory Unchanging Mean Color Unchanging

FIG. 4 is an example image 400 of a normal tracking situation.Information overlays include bounding box 402 around designated trackedobject 404, and a line 406 behind the object 404 tracing the pathhistory. The object 404 should be in full view and does not changetrajectory, moving in a constant direction. The object track 406 doesnot undergo any drastic changes within the object which allows for theuser to be fully confident in the object track 406.

Viewpoint Changes: With the normal operation mode, features and motioninformation is relatively unchanged through the track of the vehicle.The algorithms have built-in update schemes to update the feature andmotion models of the track 406 to keep tracking the object 404. When aviewpoint change occurs, the features of the object 404 do changeslightly and the motion profile may change slightly, but usually notbeyond the allocated threshold. In addition small shifts in feature andmotion confidence, viewpoint change is identified by the speed anddirection of the track 406 with respect to the imagery. The algorithmshave the ability to calculate the trajectory of the object 404 based onthe different positions of the object 404 with respect to the imagery.The combination of trajectory angle and changes in confidence within thetrack 406 will allow the characterization of viewpoint changes.

This tracking mode is helpful for the user in reorienting andreacquiring the object following a period of inattentiveness to thevideo feed. Although the tracker may be tracking the object 404adequately, the user may not recognize the object 404 any longer due tothe changing appearance of the object 404. Different viewpoints of theobject 404 may create entirely different features, which may confuse theuser if the user is not paying attention to the evolution of the objectfeatures. Therefore, it is necessary to notify the user of the changingviewpoints so that the user can direct attention to the video and updatehis own awareness of the object appearance, which in turn will assurethe user that the object tracker is continuing to follow the same object404. The characterization of the viewpoint change is shown in TABLE 4below.

TABLE 4 Information Type Characterization Feature Confidence MediumConfidence Motion Confidence High Confidence Image RegistrationConfidence High Confidence Trajectory Changing Mean Color Unchanging

FIG. 5 is an example of an image 500 having a viewpoint change of object404′ with respect to object 404 (FIG. 4). The vehicle is viewed from adifferent orientation, which may confuse the user if attention wasdirected elsewhere for a time while the orientation was changing.Notifying the user as the viewpoint is changing provides the user withan opportunity to see the object appearance evolve.

Illumination Changes: Illumination changes have similar effects onobject appearance as viewpoint changes, and they often co-occur.Illumination can change based on cloud coverage and shadowing of nearbyobjects, as well as change in object trajectory or viewpoint whichalters the lighting upon the object surface. Although the object lookssimilar, it may be sufficiently different to confuse the user and undersome conditions, may also cause the object tracker to lose the track.Illumination change can be assessed based on features (medium to lowconfidence), trajectory information (unchanging or changing), mean color(changing), and motion confidence (high).

The motion model should be fairly constant within the illuminationchange, creating a similar blob in both normal operating conditions andshadow regions. We can also see that the trajectory of the object maynot drastically change for the object. However, the mean color typicallychanges. The mean color of the object can be generated for eachdetection of each frame and can then be used to determine the colorchanges of the object. Only in an illumination change does the colormove in a certain direction. The characterization of the mode is shownin TABLE 5 below.

TABLE 5 Information Type Characterization Feature Confidence Medium/LowConfidence Motion Confidence High Confidence Image RegistrationConfidence High Confidence Trajectory Unchanging or Changing Mean ColorChanging

FIG. 6 is an example image 600 of an illumination change of object 404″with respect to object 404′ (FIG. 5). The features may change with theobject 404″, but that is due to the lowered illumination of the object404″. With illumination changes the object 404″ may continue along thesame trajectory, but the object 404″ remains still visible with changesin mean color. Although the color change may look like a partialocclusions on the object, the object's motion model still will be fairlygood with a high confidence, thus characterizing an illumination change.

Partial/Full Occlusion: An occlusion occurs when background scenery(such as a building or hill) or another object (such as trees) blocksthe view of the object of interest in whole or in part. An occlusion canbe diagnosed by the confidence values of the different parts of thealgorithm. During an occlusion, feature tracking confidence usually dipsbecause some or all of the features of the object observed in a priorframe cannot be found in the current frame. During partial occlusions,motion confidence is low but some portions of the object image motionare still visible to the tracker. During full occlusions, the motionmodel is completely gone. In all cases, a motion estimator is used toestimate the track of the object because both the feature tracking andmotion models fail to adequately find and track the object.

The motion estimation algorithm is the only suitable algorithm todetermine the trajectory of the object. But this type of occlusionreporting is time sensitive. The occlusion period can sometimes belarger than anticipated, thus confusing the tracker in the ability toreport the correct information. An occlusion that happens in a shorterduration can still be tracked, but, but as the time grows longer, theobject is more considered a lost track. The characterization of anocclusion situation is shown in the TABLE 6 below.

TABLE 6 Type Characterization Feature Confidence Low Confidence MotionConfidence Low Confidence Image Registration Confidence High ConfidenceTrajectory Only Motion Estimation Mean Color N/A

FIG. 7 is a video depiction of an image 700 of a full occlusion of theobject by trees. The feature and motion confidence dips below thethreshold and cannot be detected. The trajectory of the object can bejudged using various estimators, but the object cannot be detected whenit is occluded. For short duration occlusions, like the trees seen inthe figure, we can trust the motion estimation model to help estimatethe track beyond the occlusions. That is, the motion model will likelyresult in a reacquisition of the object once it emerges from theocclusion. However, the certainty of reacquiring the object based on themotion estimation models drops as a function of occlusion time.

Lost Track: The lost track mode is diagnosed when the features andmotion models of the object are unable to find the object after sometime. Both confidence values will be low. What differs from theocclusion mode is the duration of the lost track. This threshold betweenocclusion and lost track can be determined by the user as a temporalthreshold. In some cases, even though the tracker may lose the featuresand motion of the object, the tracker has not permanently lost the trackof the object and may still reacquire it. The characterization of themode is shown in the TABLE 7 below.

TABLE 7 Type Characterization Feature Confidence Low Confidence MotionConfidence Low Confidence Image Registration Confidence High ConfidenceTrajectory Only Motion Estimation for extend period Mean Color N/A

Failed Image Registration: Image registration is the most crucial aspectof tracking an object with a specific reference point. The imageregistration algorithm is able to provide historical data of the objecttrajectory and place it within the feature space that is available forthe user, like the GIS feature space. The tracking algorithms do notwork as well when the historical object trajectory data is lost. This isyet another situation to notify the user about when it occurs. Failedimage registration is a general notification that the image quality ispoor and the object tracker is unlikely to function well until the imagequality improves. Image registration confidence is a key variable indiagnosing the image registration failure situation. Thecharacterization of the failure mode is shown in TABLE 8 below.

TABLE 8 Type Characterization Feature Confidence Low Confidence MotionConfidence Low Confidence Image Registration Confidence Low ConfidenceTrajectory Drastic Change Mean Color N/A

FIG. 8 is an example image 800 of a failed image registration event.Within the previous images the background and object features werediscernable, with a fairly distinct tracking point. With more focusedimages the algorithm is usually able to register the points of theobject, which can be characterized by the image registration algorithm.Once the object and background features are no longer distinct, or areout of view, the algorithm is unable to register the image, thusdestroying the tracks that were previously created. There aremethodologies to discard bad image registration occurrences, but itrequires a development of frame history.

Scene Characterization Usage: The value of diagnosing and reportingthese tracking modes is in providing information to the user about thechanging nature of the visual scene. Informing the user about changes tothe scene primes the user to sample the video feed and update his ownmodel of the object appearance. Informing about shifting tracking modesthey occur also increases user awareness of scene changes that maydegrade the automated tracking, helping the user make real-timedecisions on whether to trust the object tracker or pay closer attentionto the video feed. Over time, as the user experiences sees the trackingsituation emerge under different viewing conditions, the user becomesmore aware of the performance limitations of the object tracker. Thisinformed awareness allows the user to properly calibrate trust in thealgorithm and make better decisions about when to reply upon it.

The different tracking modes or situations can be assessed using theinformation currently generated by the tracking architecture. However,not all of the tracking situations and modes are currently diagnosed andreported within the software prototype. In addition, the development ofadditional types of tracking modes is ongoing to encompass a wider rangeof viewing conditions. We are developing new algorithms that providefurther insight to the scenarios that are presented by the trackingalgorithm.

Graphical User Interfaces with Alerting:

This section describes the manner in which object tracker data isportrayed to the user. This includes portrayal of tracking confidencemeasures and also general tracking fault situations. In addition,interaction modes with the object tracker are also described, wherebythe user may opt to manually track an object of interest or delegate thetracking function to automation. The user interface has been implementedwithin a software demonstration. Diagnosis of scene changes and trackingfaults is limited to lost tracks and image processing failures.

Confidence Displays and Alerts: Human performance studies evaluating theimpact of machine confidence on trust in automation reveal differentways of portraying automation confidence. Lyons expressed confidenceinformation about an emergency landing plan as a simple percentagemeasure displayed as text. Lyons, J. B., Koltai, K. S., Ho, N. T.,Johnson, W. B., Smith, D. E., & Shively, J. R. (2016). Mercado andcolleagues used categorical levels of confidence, expressed throughcolor coding (green, yellow, red) and opacity levels of UxV icons. [17];and [18].

The present disclosure for portraying object tracker confidence displaysthe information graphically as both a continuous measure (percentage)and a categorical (color code) measure. The percentage displayed to theuser is the highest confidence level of the three object trackers. Thecolor code is green when the DRIFT feature tracker is weighted 1.0,yellow when the Moving Object Detection tracker is weighted 1.0, and redwhen the Kalman Motion Prediction object tracker is weighted 1.0. Audioalarms are also used to indicate when the automation has low confidence.Both the current and historical confidence information is displayed tothe user in several ways, including information overlays on the FMV.

FIGS. 9-11 are images 900, 1000, 1100 respectively illustrating threeexamples of object tracker confidence information 902, 1002, 1102displayed in a tracking task panel 700. There are several waysconfidence is portrayed, with some redundancy. First is the ConfidenceStrip Display, which is found above the FMV display on the Tracking TaskPanel. The amount of the strip that is filled indicates the relativedegree of object tracker confidence: more fill equals higher confidence.The Confidence Strip Display has three segmented areas corresponding tolow, medium, and high confidence. When confidence is high, the stripfills up the first (low confidence), second (medium confidence), and aportion of the third (high confidence) segment. The color of the stripis green as long as confidence is high, as shown in FIG. 9 in the top ofthe three tracking task panels. The color of the strip becomes yellowwhen the confidence level strip fills up the first (low confidence) anda portion of the second (medium confidence) segment. This is shown inthe image 1000 of FIG. 10.

Image 1100 in FIG. 11 shows the color of the strip is red whenconfidence is in the low range, and the strip only fills up a portion ofthe first segment. Another way confidence is portrayed is the color ofthe borders of the tracking task panel. The border remains white incolor when object tracker confidence is medium or high. When confidenceis low, the border of the tracking display is colored red (see bordersof bottom panel in FIG. 4). This same border color approach could befeatured around task panels on other display monitors in order to drawthe attention of the object tracker user back to the FMV. Low confidencesituations are accompanied by an auditory alarm, which begins after theobject tracker has remained in low confidence situation for at least onesecond.

A third way confidence is portrayed is on the bounding box, which theobject tracker draws around the estimated location of the object beingfollowed (a vehicle in the example shown). The color outline of the boxcorresponds to the current, relative confidence of the object tracker.Finally, notice the line drawn behind the box, which is called theConfidence Path History. The Confidence Path History shows two things:the historical estimated location of the vehicle, and the relativeconfidence level of the object tracker at each point along the estimatedvehicle path. An informal title for the line is a snail trail. As thevehicle moves it leaves behind a trail which is colored based on themomentary object tracker confidence at that time and location within theFMV. The trail remains visible as long as the relevant area on theground is visible.

In FIGS. 9-11, the respective path history 904, 1004, 1104 provides anindirect references to current and historic confidence as well. The pathhistory of the vehicle is more accurate when confidence is high(smoothly arcing path along the road) versus when confidence is mediumor low (wavy or erratic path). Thus, the relative smoothness of the pathhistory is an indirect indicator of object tracker confidence as thevehicle travels.

Tracking State Indicators and Alerts: A simple indicator box on theright side of the tracking task panel indicates the mode of tracking,which includes “No Tracking” 906, “Manual” 908, and “Auto” 906. Theseindications refer to several different states of the object tracker, andtransitions between these states and accompanying alerts representadditional innovations in human interaction with object trackers. Statetransition alerts can serve to supplement low confidence alerts.

The present disclosure supports manual (human directed) tracking, orautomated tracking. If the user wishes to display a bounding box aroundthe object, the user positions a cursor over the vehicle. Some objecttracker approaches supply a tool to expand a bounding box around anobject to follow. Other approaches will auto-detect the object when theuser clicks on the object. In the current implementation, clicking onthe object and pressing the left mouse button causes the object trackerto automatically search for and detect an object and begin creating abounding box around the object, sizing it based on the size of theobject image, and periodically resizing if the object changes size andshape in the FMV. This manual tracking bounding box is colored blue andthe tracking mode indicator states “Manual Tracking.” The bounding boxposition moves wherever the user positions the mouse in the FMV andcontinues to display as long as the user holds down the left mousebutton.

If the object tracker successfully acquires the object, it displays asecond bounding box which is colored purple. The bounding box is offsetseveral pixels from the user bounding box so they may be distinguishedfrom each other. At this point however, the user has not yet delegatedthe tracking function to the object tracker. FIG. 12 is a state diagram1200 that illustrates example user interface (UI) presentations 1210 a-crespectively for “not tracking”, “manual”, and “auto” modes of objecttracker 300 (FIG. 3). (UI) presentations 1210 a-c include trackingstates, transitions, and alerting chimes

The participant may hand-off the tracking task to the object tracker. Tohand-off to the object tracker, the participant discontinues manuallytracking by briefly pressing the space bar on the keyboard. Thisinitiates the automated object tracking mode is engaged and the trackingmode indicator boxes changes from “Manual” to “Auto.” The object trackerbounding box changes from purple to the color of the current relativeconfidence level (green, yellow, or red). The bottom screen capture inFIG. 12 (1210 c) shows the appearance of the interface during “Auto”tracking mode. At any point the participant can intervene and regaincontrol of the tracking task. To do this the participant begins manuallytracking which disengages the object tracker. The tracking modeindicator switches from “Auto” to “Manual” and the purple bounding boxis removed, and a blue bounding box is drawn.

The third state of “No Tracking” is entered either when the userdiscontinues manually tracking, or the object tracker which has beendelegated to follow an object can no longer locate the designatedobject. When the vehicle is no longer visible to the object tracker, itautomatically searches for the vehicle which is aided by a motionprediction algorithm. The object tracker predicts the estimated locationof the vehicle moment to moment based on the last known position,direction of movement, and speed. This constrains the search space forthe object tracker. This can occur under a variety of circumstances,such as when the vehicle disappears from view while driving under abridge and is not reacquired when the object emerges again.

The search for the vehicle by the object tracker can result inreacquisition of the correct vehicle, no reacquisition, or in rarercases the object tracker acquires a different vehicle visible in theFMV. When the object tracker can no longer locate the vehicle, thetracking state switches to “No Tracking” and audio chime begins to playand replay every three (3) seconds until the user manually tracks and/ordesignates again the object to follow. The switch from “Auto” trackingto “No Tracking” is the implicit diagnosis that the track has been lost.When a lost object event occurs, the participant can reset the trackerto follow the correct object again by re-designating the object. UIpresentation 1210 a is the appearance of the user interface during “NoTracking” mode.

Note that the audio alert associated with the transition to “NoTracking” is a separate, independent assessment of object trackingperformance in addition to the low confidence alerts. This is animportant distinction, because the low confidence alerts may occurbefore an object is lost, providing an early warning the object could belost. The low confidence alerts may occur at the moment the object islost, or the low confidence alert may not occur at all when the objectis lost. Likewise, it is possible the object tracker begins to follow adifferent than the one designated, and in such circumstances confidencemay be relatively high and the object tracker remains in the “Auto”tracking state.

Tracker Availability Indicators and Alerts: There are also failureswhere the object tracker cannot immediately reacquire the target evenwith user help, because the video quality is poor. Video quality can besufficiently degraded to the point the object tracker cannot functionuntil video quality improves. During these failures, the participant mayattempt a reset but the object tracker will not reacquire the object.Within traditional user interfaces, the user must repeatedly attempt todesignate the object to see whether the video quality has improvedenough for the object tracker to acquire the target. FIG. 13 is a statediagram 1300 that illustrates UI presentations 1310 a-b having indicatorbox 1320 which states “Tracker Unavailable” or “Tracker Available” basedon whether the image registration process was successful. The transitionbetween these states is also signaled by a unique chime. These twoindicators alert the participant to when the image registration hassucceeded or failed, providing the user greater insight into the natureof the object tracker performance and when the user can handoff trackingto the object tracker, and when intervention is needed.

Object Tracker Preview Mode: When the image registration process issuccessful, the user designates the target and the object trackersuccessfully acquires the target, there is still remaining thedelegation of the tracking function to the object tracker. During thisinterlude there is another innovative capability which allows the userto see the object tracker confidence before delegating the trackingfunction to the automation. This is called the “Preview Mode” and whenthis mode is engaged, the purple bounding box is instead colored thecurrent confidence level of the object tracker. The Confidence StripDisplay and Confidence Path History also populate with data. The valueto the user is to see the relative performance of the object trackerbefore delegating the tracking function.

FIG. 14 is a state diagram 1400 illustrating UI presentations 1410 a-cthat show the different tracking states when the Preview Mode is on. Todistinguish the Preview Mode from the auto tracking mode, the line forthe bounding box is dashed rather than solid. This bounding box remainsdashed until the user delegates tracking to the automation. Note,however, that if the user discontinues manually tracking (1410 a), thetracking mode will switch to “Not Tracking” and the object trackerbounding box and confidence information does not display again until theuser designates the target again. The Preview Mode can be turned on oroff by pressing a designated key. FIG. 12 shows the appearance of theuser interface in different tracking states when Preview Mode is off,and FIG. 13 shows the appearance when Preview Mode is on.

User Interface Innovation Summary: TABLE 9 summarizes the differencesbetween traditional object tracker interfaces and the proposed userinterface invention. These differences are described in the context ofdifferent types of object tracker performance situations and useractions.

TABLE 9 Object Tracker User Interface Situation Traditional ProposedUser Designates Bounding Box created by Bounding Box created by UserAround Object to Follow User Around Object Object (Blue Color) TrackingMode reads “Manual” Object Tracker No Indicators Preview Mode On/Off:Successfully   Second Bounding Box Appears around Acquires Object  Object, colored Purple Preview Mode On:    Confidence Displays Beginto    Populate    Confidence Strip Display    Confidence Path HistoryUser Bounding Box Around Object Blue Bounding Box Removed, PurpleSuccessfully Remains after User Hands- Bounding Box Remains Hands-OffOff Single Audio Chime indicating Object Tracking Successful Delegationto Object Tracker Tracking Mode reads “Auto” Confidence Displays Beginto Populate    Confidence Strip Display    Bounding Box Color   Confidence Path History Object Tracker Bounding Box Appears BoundingBox Appears Around Indicates Around Designated Object Designated ObjectRelative Relative Overlap of Bounding Relative Overlap of Bounding Boxwith Performance Box with Object Object Object Tracker ConfidenceDisplays    Confidence Strip Display    Bounding Box Color    ConfidencePath History Low Confidence Alerting    Audio Chime, repeating duringlow    confidence period    Red Border Drawn Around Interface    Panel   (Option) Red Border Around Other    Task Panels Object Tracker Lossof Bounding Box Loss of Bounding Box Loses Object Tracking ModeIndicator reads “No Tracking” Repeating Audio Chime Until UserDesignates Object Again by Manually Tracking Tracker Unable Loss ofBounding Box (if Loss of Bounding Box (if Object Tracker to ProcessVideo Object Tracker was was Following) Following) Bounding Box Does NotRemain When Bounding Box Does Not Designating Object Remain WhenDesignating Tracker Availability reads “Tracker Object Unavailable”Repeating Audio Chime Until User Designates Object Again by ManuallyTracking Tracker Able to Bounding Box Remains After Bounding Box ColorSwitches from Blue Process Video Designating Object (for manualtracking) to Purple after user designates object Tracker Availabilityreads “Tracker Available” Transition from “Tracker Unavailable” to“Tracker Available” accompanied by Audio Chime

Manner and Process of Making and Using the Present Innovation: Objecttracking algorithms are coded in Matlab and the User Interface resideswithin the Vigilant Spirit Control Station. The tracking algorithms aredemonstrated using archived video within the software prototype.

Alternatives: Object tracking algorithms could be coded alternatively inC++ or other programming language. Likewise the User Interface could becoded within a different environment. The specific representations ofconfidence information and the visual and auditory alerts could bedesigned in a variety of alternative ways. The tracking algorithms couldbe used with real-time full motion video to follow a moving object, andthe algorithms could also be used with archived video for forensicanalysis of object movements.

FIG. 15 is a flow diagram illustrating method 1500 of enabling efficienthuman collaboration with automated object tracking. In one or moreembodiments, method 1500 includes receiving, by a controller of a UAVcontrol station, full motion video of a ground scene taken by anairborne sensor (block 1502). Method 1500 includes spatially registeringfeatures of a movable object present in the ground scene (block 1504).Method 1500 includes determining motion of the movable object relativeto the ground scene (block 1506). Method 1500 includes predicting atrajectory of the movable objective relative to the ground scene (block1508). Method 1500 includes tracking the movable object based on datafusion of: (i) the spatially registered features; (ii) the determinedmotion; and (iii) the predicted trajectory of the movable object (block1510). Method 1500 includes presenting a tracking annotation on a userinterface device (block 1512).

Method 1500 includes determining a confidence value of the tracking ofthe movable object. In particular, method 1500 includes determining acurrent set of feature points of the movable object in a current portionof the full motion video (block 1514). Method 1500 includes determininga previous set of feature points of the movable object in a previousportion of the full motion video (block 1516). Method 1500 includescomparing the current set and previous set of feature points (block1518). Method 1500 includes calculating a confidence value based on arelationship between matches between the current and previous sets(block 1520). Method 1500 includes presenting a confidence indicator ofat least one of: (i) a selected color associated with a confidencelevel; and (ii) a numeric confidence value on the user interface deviceto facilitate human collaboration with object tracking (block 1522).

Method 1500 includes determining whether a visual discontinuity event ofat least one: (i) relative orientation; and (ii) lighting of the movableobject is occurring (decision block 1524). In response to determiningthat the visual continuity event is occurring, method 1500 includespresenting an alert via the user interface device to enable a human tomaintain familiarity with appearance of the movable object to expeditehuman collaboration (block 1526). In response to determining that thevisual continuity event is not occurring in decision block 1524 or afterperforming block 1526, method 1500 includes transmitting a steeringcommand to the airborne sensor to maintain the movable object within thefull motion video based at least in part on the tracking of the movableobject (block 1528). Then method 1500 returns to block 1502.

The following references that are cited above are hereby incorporated byreference in their entirety:

-   [1] Behymer, K., Kancler, D., Stansifer, C., Cone, S., &    McCloskey, M. (2013). A Cognitive Work Analysis and Visit to the    497th Intelligence, Surveillance and Reconnaissance Group—Langley    AFB, VA. Technical Report AFRL-RH-WP-TR-2013-0121, Wright Patterson    AFB, Ohio.-   [2] Jones, W., & Graley, J. (2012). AFRL Support for AFSOC PED    Analyst Needs: Available and Emerging Technology Programs to Improve    Mission Outcomes. Technical Report AFRL-RH-WP-TR-2012-0020, Wright    Patterson AFB, Ohio.-   [3] Turner, K., Stansifer, C., Stanard, T., Harrison, T., &    Lauback, D. (2013). A Cognitive Analysis of the 27th Special    Operations Group—Cannon AFB, NM. Technical Report    AFRL-RH-WP-TR-2013-0144, Wright Patterson AFB, Ohio.-   [4] McCloskey, M. J., O'Connor, M. P., Stansifer, C. M.,    Harrison, T. P., Gorham, B. D., Seibert, B. A., Horvath, G. P.,    Zelik, & Ocampo, S. M. (2015). Innovative Contested, Degraded,    Operationally-limited (CDO) Environment Analysis Characteristics,    Challenges, and Support Concepts Across the Air Force Distributed    Common Ground System (DCGS) Enterprise. Technical Report    FRL-RH-WP-TR-2015-0025, Wright Patterson AFB, Ohio.-   [5] McCloskey, M., Behymer, K., O'Connor, M., & Kuperman, G. (2012).    A Cognitive Study of the Analysis Task Performance within the 181st    Intelligence Wing. Technical Report AFRL-RH-WP-TR-2012-0166, Wright    Patterson AFB, Ohio.-   [6] For a more in-depth description of object tracking approaches,    see Smeulders, Chu, Cucchiara, Calderara, Dehghan, and Shah (2013).-   [7] Although more commonly used by Sensor Operators, FMV Analysts in    DGS could also employ object trackers to help build vehicle routing    diagrams. The object trackers can also be configured to    automatically capture images when the object performs certain    behaviors such as stop, go, and turn.-   [8] Lyons, J. B., Koltai, K. S., Ho, N. T., Johnson, W. B.,    Smith, D. E., & Shively, J. R. (2016). Engineering Trust in Complex    Automated Systems. Ergonomics in Design, 24, 13-17.-   [9] Lyons, J. B. (2013). Being transparent about transparency: A    model for human-robot interaction. In D. Sofge, D., G. J.    Kruijff, G. J., & W. F. Lawless (eds.), Trust and Automation    Systems: Papers from the AAAI spring symposium. Technical Report    SS-13-07), AAAI Press, Menlo Park, Calif.-   [10] Chen, J. Y. C., Procci, M., Boyce, J., Wright, A., Garcia, &    Barnes, M. (2014). Situation Awareness-Based Agent Transparency.    Technical Report ARL-TR-6905, Aberdeen Proving Ground, Md.-   [11] Jackovitz, K., & Asari, V. K. (2012) “Registration of region of    interest for object tracking applications in wide area motion    imagery,” Proceedings of the IEEE Computer Society Workshop on    Applied Imagery and Pattern Recognition—AIPR 2012, pp. 1-8,    Washington, D.C., 09-11 Oct. 2012. (IEEE).-   [12] Mathew, A., & Asari, V. K. (2013) “Tracking small targets in    wide area motion imagery data,” Proceedings of IS&T/SPIE    International Conference on Electronic Imaging: Video Surveillance    and Transportation Imaging Applications, San Francisco, Calif., USA,    vol. 8402, pp. 840205:1-9, 3-7 Feb. 2013. (IS&T/SPIE).-   [13] Santhaseelan, V. & Asari, V. K. (2014) “Moving object detection    and tracking in wide area motion imagery,” Wide Area Surveillance:    Real Time Motion Detection Systems, Edited by Vijayan K. Asari,    Chapter 3, Springer Book Series: Augmented Vision and Reality,    Series Editors: Riad I. Hammoud and Lawrence B. Wolff, Springer,    DOI: 10.1007/8612_2012_9, vol. 6, pp. 49-70, 2014.-   [14] Krieger, E., Sidike, P., Aspiras, T., & Asari, V. K. (2015)    “Directional ringlet intensity feature transform for tracking,” in    Image Processing (ICIP), 2015 IEEE International Conference on.    IEEE, 2015, pp. 3871-3875.-   [15] Aspiras, T., Asari, V. K., & Vasquez, J. (2014) “Gaussian    ringlet intensity distribution (grid) features for    rotation-invariant object detection in wide area motion imagery,” in    Image Processing (ICIP), 2014 IEEE International Conference on.    IEEE, 2014, pp. 2309-2313.-   [16] Mathew, A., & Asari, V. K. (2012) “Local histogram based    descriptor for object tracking in wide area motion imagery,”    International Journal of Information Processing, vol. 6, no. 4, pp.    1-12, 2012. (IJIP).-   [17] Mercado, J. E., M. A. Rupp, J. Y. C. Chen, M. J. Barnes, D.    Barber, and K. Procci. (2016). Intelligent Agent Transparency in    Human-Agent Teaming for Multi-UxV Management. Human Factors, 58(3),    401-415.-   [18] McGuirl, J. M., & Sarter, N. B. (2006). Supporting trust    calibration and the effective use of decision aids by presenting    dynamic system confidence information. Human Factors, 48 (4),    656-665.

In the preceding detailed description of exemplary embodiments of thedisclosure, specific exemplary embodiments in which the disclosure maybe practiced are described in sufficient detail to enable those skilledin the art to practice the disclosed embodiments. For example, specificdetails such as specific method orders, structures, elements, andconnections have been presented herein. However, it is to be understoodthat the specific details presented need not be utilized to practiceembodiments of the present disclosure. It is also to be understood thatother embodiments may be utilized and that logical, architectural,programmatic, mechanical, electrical and other changes may be madewithout departing from general scope of the disclosure. The followingdetailed description is, therefore, not to be taken in a limiting sense,and the scope of the present disclosure is defined by the appendedclaims and equivalents thereof.

References within the specification to “one embodiment,” “anembodiment,” “embodiments”, or “one or more embodiments” are intended toindicate that a particular feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment of the present disclosure. The appearance of such phrases invarious places within the specification are not necessarily allreferring to the same embodiment, nor are separate or alternativeembodiments mutually exclusive of other embodiments. Further, variousfeatures are described which may be exhibited by some embodiments andnot by others. Similarly, various requirements are described which maybe requirements for some embodiments but not other embodiments.

It is understood that the use of specific component, device and/orparameter names and/or corresponding acronyms thereof, such as those ofthe executing utility, logic, and/or firmware described herein, are forexample only and not meant to imply any limitations on the describedembodiments. The embodiments may thus be described with differentnomenclature and/or terminology utilized to describe the components,devices, parameters, methods and/or functions herein, withoutlimitation. References to any specific protocol or proprietary name indescribing one or more elements, features or concepts of the embodimentsare provided solely as examples of one implementation, and suchreferences do not limit the extension of the claimed embodiments toembodiments in which different element, feature, protocol, or conceptnames are utilized. Thus, each term utilized herein is to be given itsbroadest interpretation given the context in which that terms isutilized.

While the disclosure has been described with reference to exemplaryembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the scope of the disclosure. Inaddition, many modifications may be made to adapt a particular system,device or component thereof to the teachings of the disclosure withoutdeparting from the essential scope thereof. Therefore, it is intendedthat the disclosure not be limited to the particular embodimentsdisclosed for carrying out this disclosure, but that the disclosure willinclude all embodiments falling within the scope of the appended claims.Moreover, the use of the terms first, second, etc. do not denote anyorder or importance, but rather the terms first, second, etc. are usedto distinguish one element from another.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The description of the present disclosure has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the disclosure in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope of the disclosure. Thedescribed embodiments were chosen and described in order to best explainthe principles of the disclosure and the practical application, and toenable others of ordinary skill in the art to understand the disclosurefor various embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method of enabling efficient humancollaboration with automated object tracking, the method comprising:receiving full motion video of a ground scene taken by an airbornesensor; spatially registering features of a movable object present inthe ground scene; determining motion of the movable object relative tothe ground scene; predicting a trajectory of the movable object relativeto the ground scene; tracking the movable object based on data fusionof: (i) the spatially registered features; (ii) the determined motion;and (iii) the predicted trajectory of the movable object; presenting atracking annotation on a user interface device; determining a confidencevalue of the tracking of the movable object; presenting a currentconfidence indicator and at least one of historical and projected futureconfidence in performance on the user interface device to facilitatehuman collaboration with object tracking without over- andunder-reliance on the object tracker; determining whether the confidencevalue is below a minimum confidence threshold; and in response todetermining that the confidence is below the minimum confidencethreshold: determining whether the confidence level is below the minimumconfidence threshold based on one or more contributing factors from agroup consisting of: (i) view change; (ii) illumination change; (iii)partial/full occlusion; (iv) lost track; and (v) failed imageregistration; and presenting the one or more contributing factors viathe user interface device to help the user develop a deeperunderstanding over time of the performance envelope of the objecttracker under various viewing situations.
 2. The method of claim 1,further comprising transmitting a steering command to the airbornesensor to maintain the movable object within the full motion video basedat least in part on the tracking of the movable object.
 3. The method ofclaim 2, further comprising: determining whether a visual discontinuityevent of at least one: (i) relative orientation; and (ii) lighting ofthe movable object is occurring for the movable object that remainswithin a field of view of the airborne sensor; and in response todetermining that the visual continuity event is occurring, presenting analert via the user interface device, the alert capable of beingperceived by a user that is not paying attention to the ground scene toenable a human to maintain familiarity with appearance of the movableobject to expedite human collaboration.
 4. The method of claim 3,wherein the alert comprises a confidence strip display that is presentedon the user interface device outside of the ground scene.
 5. The methodof claim 3, wherein the alert comprises an auditory alarm that is anearly warning that the movable object could be lost by the objecttracker in the immediate future.
 6. The method of claim 3, wherein thealert comprises a confidence path history presented on the groundhistorical based on historical estimated locations of the movableobject.
 7. The method of claim 3, further comprising: presenting thefull motion view and tracking annotation on a first display of a controlstation; and presenting the alert on a second display of the controlstation.
 8. The method of claim 1, wherein presenting the confidenceindicator comprises presenting at least one of: (i) a selected colorassociated with a confidence level; and (ii) a numeric confidence value.9. The method of claim 1, wherein determining the confidence valuecomprises: determining a current set of feature points of the movableobject in a current portion of the full motion video; determining aprevious set of feature points of the movable object in a previousportion of the full motion video; comparing the current set and previousset of feature points; and calculating the confidence value based on arelationship between matches between the current and previous sets. 10.The method of claim 1, wherein determining and presenting the confidencevalue of the tracking of the movable object comprises: determiningwhether a fault has occurred that presents automated tracking; and inresponse to determining that the fault has occurred, presenting theconfidence indicator comprising the fault occurrence on the userinterface device to expedite human collaboration with object tracking.11. A system that enables efficient human collaboration with automatedobject tracking, the system comprising: an aerial vehicle having anairborne sensor; and a control station communicatively coupled to theaerial vehicle to: receive full motion video of a ground scene taken bythe airborne sensor; present the ground scene on a user interfacedevice; receive a user input that indicates the movable object on theground annotate a manual tracking annotation on the user interfacedevice that corresponds to the user input; searching for a movableobject within a portion of the ground scene corresponding to the userinput; spatially register features of the movable object present in theground scene; determine motion of the movable object relative to theground scene; predict a trajectory of the movable object relative to theground scene; track the movable object based on data fusion of: (i) thespatially registered features; (ii) the determined motion; and (iii) thepredicted trajectory of the movable object; in response to determiningthat automatic tracking is available: present a tracking annotation onthe user interface device that is offset from the manual trackingannotation; determine a confidence value of the tracking of the movableobject; present a confidence indicator on the user interface device tofacilitate human collaboration with object tracking; and switch frommanual object tracking to automatic object tracking in response to auser input; determine whether the confidence value is below a minimumconfidence threshold; and in response to determining that the confidenceis below the minimum confidence threshold: determine whether theconfidence level is below the minimum confidence threshold based on oneor more contributing factors from a group consisting of: (i) viewchange; (ii) illumination change; (iii) partial/full occlusion; (iv)lost track; and (v) failed image registration; and present the one ormore contributing factors via the user interface device to help the userdevelop a deeper understanding over time of the performance envelope ofthe object tracker under various viewing situations.
 12. The system ofclaim 11, wherein the control station transmits a steering command tothe airborne sensor to maintain the movable object within the fullmotion video based at least in part on the tracking of the movableobject.
 13. The system of claim 11, wherein the control station:determines whether a visual discontinuity event of at least one: (i)relative orientation; and (ii) lighting of the movable object isoccurring; and in response to determining that the visual continuityevent is occurring, presents an alert via the user interface device toenable a human to maintain familiarity with appearance of the movableobject to expedite human collaboration.
 14. The system of claim 11,wherein control station presents the confidence indicator via at leastone of: (i) a selected color associated with a confidence level; and(ii) a numeric confidence value.
 15. The system of claim 11, wherein thecontrol station determines the confidence value by: determining acurrent set of feature points of the movable object in a current portionof the full motion video; determining a previous set of feature pointsof the movable object in a previous portion of the full motion video;comparing the current set and previous set of feature points; andcalculating the confidence value based on a relationship between matchesbetween the current and previous sets.
 16. The system of claim 11,wherein the control station determines and present the confidence valueof the tracking of the movable object by: determining whether a faulthas occurred that presents automated tracking; and in response todetermining that the fault has occurred, presenting the confidenceindicator comprising the fault occurrence on the user interface deviceto expedite human collaboration with object tracking.
 17. A controlstation that enables efficient human collaboration with automated objecttracking, the control station comprising: a network interfacecommunicatively coupled to an aerial vehicle having an airborne sensor;and a controller communicatively coupled to the network interface andthat enables the control station to: receive full motion video of aground scene taken by the airborne sensor; spatially register featuresof a movable object present in the ground scene; determine motion of themovable object relative to the ground scene; predict a trajectory of themovable object relative to the ground scene; track the movable objectbased on data fusion of: (i) the spatially registered features; (ii) thedetermined motion; and (iii) the predicted trajectory of the movableobject; present a tracking annotation on a user interface device;determine a confidence value of the tracking of the movable object;present a current confidence indicator and at least one of historicaland projected future confidence in performance on the user interfacedevice to facilitate human collaboration with object tracking withoutover- and under-reliance on the object tracker; determine whether theconfidence value is below a minimum confidence threshold; and inresponse to determining that the confidence is below the minimumconfidence threshold: determine whether the confidence level is belowthe minimum confidence threshold based on one or more contributingfactors from a group consisting of: (i) view change; (ii) illuminationchange; (iii) partial/full occlusion; (iv) lost track; and (v) failedimage registration; and present the one or more contributing factors viathe user interface device to help the user develop a deeperunderstanding over time of the performance envelope of the objecttracker under various viewing situations.
 18. The control station ofclaim 17, wherein the controller transmits a steering command to theairborne sensor to maintain the movable object within the full motionvideo based at least in part on the tracking of the movable object. 19.The control station of claim 18, wherein the controller: determineswhether a visual discontinuity event of at least one: (i) relativeorientation; and (ii) lighting of the movable object is occurring forthe movable object that remains within a field of view of the airbornesensor; and in response to determining that the visual continuity eventis occurring, presents an alert via the user interface device, the alertcapable of being perceived by a user that is not paying attention to theground scene to enable a human to maintain familiarity with appearanceof the movable object to expedite human collaboration.
 20. The controlstation of claim 17, wherein controller presents the confidenceindicator via at least one of: (i) a selected color associated with aconfidence level; and (ii) a numeric confidence value.
 21. The controlstation of claim 17, wherein the controller determines the confidencevalue by: determining a current set of feature points of the movableobject in a current portion of the full motion video; determining aprevious set of feature points of the movable object in a previousportion of the full motion video; comparing the current set and previousset of feature points; and calculating the confidence value based on arelationship between matches between the current and previous sets. 22.The control station of claim 17, wherein the controller determines andpresents the confidence value of the tracking of the movable object by:determining whether a fault has occurred that presents automatedtracking; and in response to determining that the fault has occurred,presenting the confidence indicator comprising the fault occurrence onthe user interface device to expedite human collaboration with objecttracking.